Repository navigation
Conversation
Add --graph-output FILE and --graph-output-fmt {gam,gaf} to vg giraffe.
The alignments giraffe emits also go, before any surjection, through the
emitter that -o GAM or -o GAF would make, to FILE. With -o BAM, one mapping
pass then gives both a linear-reference BAM and the GAM that vg pack and
vg call need, instead of mapping the reads twice.
A new TeeAlignmentEmitter owns both emitters and gives each every batch;
the primary one gets a copy, since an emitter may move from its batch.
Tests check, against separate runs, that the SAM is unchanged, that the
GAM holds the same records (apart from time_used) and that the GAF is the
same file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adamnovak
left a comment
There was a problem hiding this comment.
I think this feature is a good idea. We didn't need all the benchmarking; it's pretty clear that writing to GAM and then loading up the graph again and passing over everything again to surject it is slower.
I think we might need to think more about how to explain and structure this though. The PR is thinking of it as Giraffe usually outputting linear alignments, and then in that case you can now additionally output graph alignments, with a file and a format.
But I think we usually present Giraffe as a tool for outputting graph alignments, and then you can optionally change the output to a linear format. If I'm a user expecting graph output, and I see:
--graph-output FILE also write the graph alignments to FILE, as
-o GAM/GAF would, before any surjection
Then I might wonder if I'm somehow not supposed to expect graph output by default, and if I need either this option or -o to get it. If I want to construct a command line that outputs both GAF and GAM, I can do it with the new flags, but then one of them is the "graph" output and one of them isn't.
I think what we should probably do is move another level up in abstraction, so everything becomes more natural. Instead of output to standard out and also up to exactly one file that can only contain un-surjected alignments, we should have an output file option. We can use the extension on the output file to determine the format, so it can support graph and surjected alignment. And we can have any number of them, so if you want GAF, GAM, and BAM, you can do that. It could replace the output file prefix option; when we need to make multiple files for the old parameter grid search stuff, we can stick the extra name component before the last extension of each file. And if you specify any of these, you won't get standard out output, and -o should be disallowed. To implement all this, the TeeAlignmentEmitter would be extended to any number of downstream emitters.
I'll see if I can get this to look more like that.
| if (!graph_output_name.empty() && !output_basename.empty()) { | ||
| logger.error() << "A graph output file (--graph-output) cannot be used " | ||
| << "with an output basename (--output-basename)." << std::endl; | ||
| } |
There was a problem hiding this comment.
Why shouldn't this work? It makes sense in the one-setting-combination case.
| * A TeeAlignmentEmitter is an AlignmentEmitter that hands every batch of | ||
| * alignments to two emitters it owns: a primary one and a copy one. | ||
| * | ||
| * Both emitters receive the same alignments in the same batches, so each one | ||
| * writes what it would have written if it had been the only emitter. This lets | ||
| * one mapping run write, for example, surjected SAM/BAM/CRAM through the | ||
| * primary emitter and the graph alignments as GAM or GAF through the copy | ||
| * emitter, instead of mapping the reads twice. | ||
| * | ||
| * The primary emitter gets a copy of each batch, and the copy emitter gets the | ||
| * batch itself, since an emitter may move from or change what it is given. | ||
| * Like any AlignmentEmitter, a TeeAlignmentEmitter may be called from several | ||
| * threads at once, and relies on the two emitters it owns to be safe for that. |
There was a problem hiding this comment.
This sounds a lot like unrefined Anthropic Claude output:
- It restates invariants of the base class only to tell us that of course they continue to apply.
- It defines the class in terms of the user-facing feature it is going to enable instead of as an abstraction to be understood in isolation.
- It doesn't put the cover on the abstraction, so the distinct concepts of "primary emitter" and "copy emitter" leak out to take up slots in the class user's brain when they don't need to.
- It has a lot of unnecessarily specific qualifiers (we hand "every batch" of alignments to the other emitters, as if a class that decided not to handle some batches was a live possibility).
- It's 3 paragraphs long when
An AlignmentEmitter that emits alignments with two other AlignmentEmitters.would suffice.
Now, this is in artifacts/, where unrefined Anthropic Claude output indeed belongs. But it's such a simple job for the class that I don't know why we wouldn't just fix it up and promote it.
Replace --graph-output and --graph-output-fmt with a repeatable --output-file, which takes the place of standard output. With ranges, each file name gets the settings slug before its extension, as --output-basename names do. Move TeeAlignmentEmitter to src/, taking any number of destinations. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Leave -o unset unless given, check output file extensions against the formats -o takes, and test standard output, one output file and several. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Refuse two outputs in the same format, and check -o and output file extensions together after parsing. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Keep each output's format and file in an unordered map, check output files for writability as they are parsed, and give each output file its own column in the --report-name TSV. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Only SAM, BAM and CRAM get the positional graph, so named coordinates work beside surjected output or correctness tracking. Check output files can be written in one place after parsing, sharpen comments, and test the report, the file cleanup and each refusal's message. Co-authored-by: Adam Novak <anovak@soe.ucsc.edu> Assisted-by: Claude Code:claude-opus-5-5
Reduce tests for error messages and many combinations of output formats. Rework logging emitter to be simpler. Rewrite error messages. Rewrite flag descriptions. Steal -O flag for new output file option. Redo comments for TeeAlignmentEmitter. Rework output file writability detection to not need to make and then delete files.
|
OK, I've reworked this to replace I'm stealing |
What this does
vg giraffegains two output options:With
-o BAM(or SAM/CRAM), one mapping pass then writes both outputs:-o gam/-o gafwould build (same graph, same flags, GAF header lines included).It works with any
-o; for example-o gam --graph-output x.gaf --graph-output-fmt gafwrites both GAM and GAF. It is refused with--output-basename(grid search writes one file per combination) and with-n.Implementation
artifacts/tee_alignment_emitter.{hpp,cpp}: a newTeeAlignmentEmitter. It is anAlignmentEmitterthat owns two emitters and hands every batch (singles, mapped singles, pairs, mapped pairs, extra messages) to both. The primary emitter gets a copy of each batch, because an emitter may move from what it is given. The copy emitter gets the batch itself.giraffe_main.cpp: wraps the existing emitter in aTeeAlignmentEmitterwhen--graph-outputis set. With no--graph-output, nothing changes.artifacts/: the new files are there, perCLAUDE.md, since they were written with Claude Code; I have read them. I did not run thelint-new-code(SonarLint) step, because there is no Java runtime on this machine.Use case and measured benefit
Our pipeline uses one mapping pass for two consumers:
vg pack→vg callstructural-variant genotyping.Without this, giraffe runs twice per sample (
-o bam, then-o gam), or the GAM is surjected afterwards withvg surject.vg surjectis not the same surjection as giraffe's internal one: the band padding, the proper-pair limit and the pairing logic all differ. On our data, 0.08% of BAM records differed.We measured an earlier version of this change. It was a local patch on v1.75.1, the same tee in front of the surjecting emitter, but switched by an environment variable and without the batch copy. The run was a 30× HG002 short-read genome (681,708,020 reads) on the HPRC v1.1 CHM13 graph, on one 64-vCPU Graviton box:
vg packvg pack-o gamrun. Thevg callVCF records were identical.This version adds one copy of each batch for the primary emitter. I have not re-measured it whole-genome. On the 1 kb
xtest graph, 200,000 simulated pairs at-t 4(Docker on an arm64 Mac, median of 3 runs), mapping costs almost nothing, so this shows the serialization cost, not the production ratio:-o BAM-o BAM --graph-output x.gam-o GAMalone (the second pass this replaces)Tests
test/t/50_vg_giraffe.t, paired reads on thextest graph at-t 1, against separate runs:-o gamrun.time_usedis set to null first, since it records each read's mapping time and differs between any two runs. The embeddedPARAMS_JSONis also the same;-o gafrun;--graph-outputwith--output-basenameis refused.artifacts/unittest/tee_alignment_emitter.cpp: both emitters receive every alignment and extra message, even when the primary clears what it is given; null emitters are refused.Local results, master at e0d5918 plus this branch, built in vg's Dockerfile
packagesstage (Ubuntu 26.04, arm64):prove t/50_vg_giraffe.t: 96/96 pass;vg test "[emitter]": 4 assertions in 2 test cases pass;scripts/lint.py: clean;lint_prose.py --base HEAD: clean.